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ABSTRACT 


In  order  to  assure  that  NILECJ  equipment  standards 
have  the  impact  intended,  a  Compliance  Testing  and 
Laboratory  Accreditation  program  is  needed  to 
establish  which  items  available  on  the  market  do,  in 
fact,  meet  the  requirements  of  the  standards.  This 
report  contains  recommendations  for  such  a  NILECJ 
program.     In  brief,   the  proposed  program  would  (a) 
result  in  a  body  of  qualification  and  acceptance  test 
data,    (b)   establish  a  list  of  testing  laboratories 
competent  to  perform  these  tests,  and   (c)   set  up  a 
"compliance  information  system"  for  the  dissemination 
of  this  information  to  officials  in  the  criminal  justice 
system. 

Key  words:     Acceptance  testing;  compliance  testing; 
laboratory  evaluation;  performance  testing;  qualified 
products  lists;  testing. 
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INTRODUCTION 

The  National  Institute  of  Law  Enforcement  and  Criminal 
Justice   (NILECJ)   of  the  Department  of  Justice  is  assisting  law 
enforcement  agencies  in  the  procurement  and  effective  use  of 
the  best  equipment  that  present  technology  can  offer.  One 
means  to  that  end  is  the  development  of  performance  standards 
for  law  enforcement  equipment,  a  task  assigned  to  the  Law 
Enforcement  Standards  Laboratory   (LESL)   of  the  National  Bureau 
of  Standards.     These  performance  standards  can  be  referenced  in 
purchase  documents;  they  serve  as  objective  descriptions  of 
wha-  the  user  wants  and  what  the  supplier  agrees  to  give  him. 
They  can  also  help  guide  manufacturers  in  the  design  of 
equipment  of  high  utility. 

The  development  and  dissemination  of  standards,  however, 
are  only  steps  in  the  procurement  process.     The  ultimate 
benefits  to  the  law  enforcement  community,  to  the  public,  and 
to  the  manufacturers  cannot  be  realized  until  a  means  is 
established  to  ascertain  which  hardware  available  on  the  market 
does,  in  fact,  meet  the  requirements  of  the  standards. 

Lester  C.  Thurow,  professor  of  economics  and  management  at 
the  Massachusetts  Institute  of  Technology,  has  written  of  this 
need  in  a  private  communication. 

"Without  testing  and  inspection,  there  is  the  classic 
economic  problem  of  bad  money  driving  out  good.     To  obtain  an 
order,  a  company  must  submit  the  lowest  bid.     One  of  the 
easiest  ways  to  submit  a  lower  bid  is  to  lower  the  quality  of 
the  equipment  being  sold.     The  initial  quality  cuts  are  small, 
but  competitive  pressures  force  an  escalation  of  this  process 
until  the  equipment  no  longer  provides  the  services  desired. 

"With  simple  goods  this  problem  can  be  avoided,   since  the 
customer  can  quickly  determine  that  he  is  being  sold  a  good 
without  the  desired  performance  characteristic.  VJith 
complicated  equipment,  there  is  no  simple  or  easy  way  to 
determine  whether  the  good  is  up  to  standards.     To  know  whether 
the  equipment  is  good  or  bad  requires  technical  testing,  both 
initially  to  insure  that  equipment  can  meet  performance 
standards  and  subsequently  to  insure  that  it  does. 
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"This  testing  is  in  the  long-run  interest  of  manufacturers 
(although  it  may  not  be  in  the  interest  of  any  particular 
manufacturer)    since  it  allows  high  quality  equipment 
manufacturers  to  stay  in  business.     Without  testing,   they  will 
be  forced  to  lower  quality  standards  by  the  competitive 
pressures  of  the  market  place  or  go  out  of  business." 

This  report  makes  recommendations  to  NILECJ  for  the 
establishment  of  a  compliance  testing  system  for  equipment  used 
by  the  criminal  justice  system.     This  "Compliance  Testing 
System"  would   (a)  make  available  to  law  enforcement  officials 
impartial  information  concerning  compliance  with  NILECJ 
standards,  and   (b)   enable  manufacturers  to  avoid  the  time  and 
expense  of  convincing  individually  hundreds  of  police 
departments  that  their  products  are  capable  of  satisfying 
minimum  performance  requirements. 

Compliance*  Testing  Mechanisms 

There  are  several  alternatives,  not  necessarily  exclusive 
of  one  another,   for  ascertaining  the  compliance  of  law 
enforcement  equipment  with  the  K'lLECJ  standards.     First,  some 
larger  police  departments  have  test  and  evaluation  divisions 
responsible  for  determining  the  utility  of  new  products. 
However,  most  departments  do  not  have  in-house  capabilities  for 
performing  compliance  tests  in  accordance  with  the  NILECJ 
standards . 

Secondly,   inspectors  can  monitor  the  manufacturing  process 
at  the  plant.     This  "on-site  inspection"  can  be  used  to  monitor 
quality  control.     It  is  being  used  for  some  food  inspection  and 
military  procurement  programs,  but  it  is  expensive  and  is  not 
considered  to  be  a  viable  alternative  for  NILECJ. 

Thirdly,  manufacturers  can  be  asked  to  certify  that  their 
equipment  meets  the  requirements  of  the  appropriate  standard, 
and  be  held  legally  accountable  for  their  certification. 
However,  reliance  solely  on  manufacturer  certification 
generally  is  not  satisfactory.     Since  transgressions  must  be 
redressed  after  the  fact,  usually  by  time-consuming  court 
action,   there  is  little  protection  against  companies  that  have 
no  reputation  to  uphold  and  are  able  to  go  out  of  business 
after  a  short  period  of  operation.     Even  for  more  reputable 
companies,   the  pressure  of  realizing  a  profit  too  frequently 
tempts  manufacturers  into  compromises  of  product  quality  if 


*These  and  other  terms  relating  to  product  standards, 
product  testing  and  laboratory  accreditation  are  defined  in 
the  glossary  on  pages  13  to  18. 
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routine  independent  checks  are  not  made.     See,  for  example, 
"The  Aircraft  Brake  Scandal,"  in  the  April  1972  issue  of  Harper 
Magazine,  which  relates  a  tale  of  falsified  laboratory  test 
results  and  fraudulent  claims  by  a  manufacturer  having  trouble 
fulfilling  a  government  contract.     Manufacturers  may  of  course 
do  their  own  product  testing.     This  certainly  should  be 
encouraged,  as  long  as  the  results  are  intended  primarily  for 
the  manufacturer's  own  use. 

Another  approach  is  to  have  the  product  manufacturer 
certify  and  an  independent  testing  laboratory  verify.     In  this 
way,  the  manufacturer  retains  the  legal  responsibility  for  the 
quality  of  his  product.     Once  the  product  is  found  by  the 
testing  laboratory  to  be  in  compliance  with  the  appropriate 
standard,  the  name  of  both  the  product  and  its  manufacturer  can 
be  placed  on  a  qualified  products  list   (QPL) .     As  long  as  his 
product  remains  on  the  published  QPL,  the  manufacturer  is 
obligated  to  make  that  product  in  compliance  with  the  product 
standard. 

Two  types  of  product  compliance  testing  are  of  interest. 
Product  qualification  tests  are  performed  in  advance  of  and 
independent  of  any  specific  procurement  action,  for  the  purpose 
of  establishing  a  QPL.     The  fact  that  a  product  has  been  tested 
and  placed  on  a  QPL  means  that  the  particular  model  meets  the 
requirements  of  the  standard.     The  inclusion  of  a  product  on  a 
QPL  does  not,  however,  guarantee  the  acceptability  of  every 
individual  lot  or  shipment.     This  assurance  can  be  obtained 
from  product  acceptance  tests ,  which  are  performed  to  determine 
the  acceptability  of  delivered  items  which  have  been  purchased 
under  a  contract  requiring  compliance  with  the  appropriate 
standard. 

The  Compliance  Testing  System  must  be  based  on  technical 
competence  and  integrity  if  it  is  to  enjoy  any  significant 
degree  of  acceptance.     For  example,  the  testing  laboratories 
must  be  independent  laboratories.     Independent  laboratories  can 
be  commercial  testing  laboratories   (there  are  several  thousand 
in  the  United  States) ,  non-profit  institution  laboratories, 
government  laboratories,  or  university  laboratories.  By 
definition,  an  independent  laboratory  cannot  be  connected  with 
the  manufacturer  of  the  product,  cannot  be  involved  in  the 
promotion  of  the  product,  and  cannot  have  an  inordinate  (for 
purposes  considered  here,  greater  than  10  percent)   amount  of 
its  income  derived  from  a  company  which  manufactures  a  product 
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of  the  type  under  test.     A  corporate  or  captive  laboratory 
which  is  controlled  by  a  manufacturer  or  distributor  is 
obviously  not  an  unbiased,  independent  laboratory,  even  though 
it  may  be  a  very  competent  one;  however,  the  laboratory 
accreditation  program  described  later  can  benefit  captive  as 
well  as  independent  laboratories. 

Other  safeguards  could  readily  be  incorporated  into  the 
program.     Renewal  qualification  tests  will  be  required,  to 
assure  that  changes  in  product  design  have  not  been  made, 
either  intentionally  or  unintentionally.     When  possible, 
manufacturers  should  be  required  to  select  different  testing 
laboratories  for  the  original  and  the  renewal  qualification 
testing  of  a  product.     In  addition,  the  results  of  any 
acceptance  tests  of  a  given  product  could  be  compared  for 
consistency  with  the  qualification  test  results.     Products  for 
which  comparatively  large  amounts  of  acceptance  test  results 
were  available  should  require  less  frequent  renewal  testing  to 
stay  on  the  QPL  than  those  products  for  which  few  acceptance 
test  data  were  available. 

The  samples  tested  in  qualification  tests  must  be 
representative  of  those  produced  in  production  runs.     If  sales 
volume  is  large  enough,  this  is  best  accomplished  by  having  the 
samples  purchased  anonymously  through  a  regular  retail 
supplier.     However,  purchasing  the  test  samples  may,  in  some 
cases,  add  significantly  to  the  cost  of  the  compliance  testing. 
There  may  also  be  some  legal  obligation  to  purchase  and  test 
all  available  brands.     This  could  lead  to  difficulties; 
conceivably,  a  manufacturer  could  insist  that  MILECJ  purchase  a 
product  which  obviously  would  not  meet  the  requirements  of  the 
standard,  and  he  could  demand  an  exorbitant  price. 

An  alternative  is  to  invite  all  manufacturers  of  law 
enforcement  equipment  for  which  L'lLECJ  standards  have  been 
promulgated  to  submit  their  products  for  qualification  testing, 
rianuf acturers  would  be  required  to  certify  that  the  submitted 
products  were  representative  of  their  production  in  all 
significant  respects.     Significant  differences  between 
qualification  and  acceptance  test  results  would  then  indicate  a 
violation  of  that  product's  certification  or  trouble  with  one 
of  the  testing  laboratories,  either  of  which  would  require 
remedial  action. 


Tlie  methods  used  for  qualification  and  acceptance  testing 
must  be  identical.     However,  economic  considerations  may 
dictate  that  acceptance  testing  be  less  thorough  than 
qualification  testing.     In  that  case,   the  methods  for 
acceptance  testing  might  consist  of  a  subset  of  the  test 
methods  for  qualification  testing.     LESL  could  recommend  the 
appropriate  subset.     More  than  one  subset  may  have  to  be 
available  to  accommodate  different  local  economic 
circumstances.     LESL  should  make  certain  that  the  purchaser  is 
properly  advised  by  the  qualified  testing  laboratory  of  the 
reduced  quality  assurance  to  be  expected  for  a  given  subset  of 
test  methods.     Advice  should  also  be  available  concerning 
appropriate  sampling  plans. 

Copies  of  all  test  reports  from  participating  laboratories 
should  be  submitted  to  the  compliance  program  administrators, 
whether  the  items  under  test  pass  or  fail.     This  procedure  will 
preclude  reporting  only  successful  test  results,  and  will 
prevent  a  manufacturer  from  repeatedly  having  an  unacceptable 
product  tested  until,  by  the  laws  of  probability,  it  is 
eventually  passed.     To  guard  against  incomplete  information 
being  sent,   the  complete  laboratory  job  file  should  be  open  to 
inspection  by  the  program  administrators. 

The  possibility  of  authorizing  product  manufacturers  to 
affix  a  NILECJ  label  or  logo  to  products  listed  on  a  QPL  has 
been  considered.     Use  of  such  a  logo  would  have  value  in 
bringing  the  NILECJ  Equipment  Program  to  the  attention  of  law 
enforcement  officials.     However,  the  use  of  a  logo  does  have 
the  following  disadvantages; 

(a)  A  product  may  be  tested  and  found  acceptable  for  one 
application,  but  not  another.     For  example,  a  particular  police 
helmet  may  give  satisfactory  protection  as  a  motorcycle  crash 
helmet,  but  not  as  a  ballistic  helmet.     Specific  product 
certification  cannot  be  communicated  with  a  logo  as  it  can  be 
with  a  QPL.     A  user  who  sees  the  logo  but  does  not  check  the 
QPL  can  develop  an  unjustified  confidence  in  the  utility  of  his 
equipment.     Careful  labeling  could  minimize  this  problem. 

(b)  Logos  are  usually  intended  for  equipment  that  has 
passed  qualification  tests.     A  logo  may,  however,   lead  a  user 
to  assume  that  an  item  has  passed  both  qualification  and 
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acceptance  tests.     That  is,  he  may  improperly  assume  that  the 
logo  guarantees  the  quality  of  the  shipment  he  receives. 

(c)   Comparatively  speaking,  potential  enforcement  problems 
for  QPL's  should  be  smaller  than  for  logos.     That  is,  entries 
on  a  QPL  would  clearly  be  under  the  control  of  NILECJ,  but 
there  may  be  difficulties  in  forcing  a  manufacturer  to  stop 
affixing  the  logo  to  products  which  no  longer  comply  with  the 
standard,  or  in  neutralizing  the  logo  on  those  already  in 
distribution.     There  is,  of  course,   the  similar  problem 
involved  in  assuring  that  a  procurement  decision  is  based  on  a 
current,  not  an  obsolete,  QPL. 

Laboratory  Lvaluation  and  Accreditation 

Testing  laboratories  other  than  NBS  will  have  to  be 
involved  with  compliance  testing.     Accreditation  of  these 
testing  laboratories  will  be  an  important  part  of  the 
Compliance  Testing  System.     Only  accredited  laboratories  should 
be  eligible  for  selection  to  perform  qualification  testing,  and 
only  accredited  laboratories  should  be  recommended  to  police 
officials  who  are  selecting  a  laboratory  for  acceptance 
testing .        .  . 

Laboratory  accreditation  should  be  distinguished  from 
laboratory  certification.     Certification  implies  a  warranty  or 
guarantee  of  a  laboratory's  performance.     There  is  no  valid 
approach  to  examining  a  laboratory  so  that  guarantees  of  future 
performance  can  be  issued.     Accreditation  resulting  from 
laboratory  evaluation  does  not  ensure  the  quality  of  future 
output.     Rather,  it  indicates  that  there  is  no  apparent  reason 
why  the  laboratory  cannot  perform  a  function  adequately.  The 
only  realistic  goal  of  laboratory  evaluation  is  accreditation, 
not  certification.     However,  laboratory  performance  can  be 
monitored  after  the  initial  accreditation   (a)  by  comparing  data 
sent  to  the  Compliance  Information  System  from  different 
laboratories  testing  the  same  product,    (b)  via  occasional 
unannounced  examinations  of  accredited  laboratories  and   (c)  by 
requiring  that  accreditation  be  renewed  every  few  years. 
Laboratory  accreditation  can  be  revoked  in  cases  of 
misrepresentation  of  information  or  unsatisfactory  performance. 

Accreditation  would  be  granted  for  those  testing  functions 
of  laboratories  that  meet  pre-stated  criteria.  These  criteria 
will,   for  the  most  part,  relate  to  the  methods  of  test  in 


specific  x.ILLCJ  standarus,   although  sone  criteria  such  as ^ 
raanacjerial  independence  froia  luanuf acturers  and  vendors  and 
records  keeping  will  have  general  applicability.     The  Law 
Lnforceraent  Standards  Laboratory  at  the  "National  Bureau  of 
Standards  is  currently  developing  laboratory  evaluation 
criteria  for  the  first  few  performance  standards  promulgated  by 
iilLLCJ  . 

Laboratory  accreditation  criteria  and  evaluations  can  be 
classified  as  either  per f ornance-based  or  non-per f ornance- 
based.     i.on-perf ormance-based  criteria  can  relate,   for  example, 
to  the  cleanliness  of  laboratory  equipment  or  facilities,   or  to 
the  academic  backgrounds  of  personnel.  Performance-based 
criteria,   on  the  other  hand,   relate  to  the  actual  results  of 
tests  on  controlled  samples  by  the  laboratory  under 
examination . 

The  objective  of  non-performance-based  evaluation  is  to 
predict  performance.     Since  performance-based  evaluations  are 
intierently  more  direct,   they  will  be  used  to  the  greatest 
extent  practical. 

Laboratory  accreditation  criteria  can  also  be  classified  as 
either  technical  or  nontechnical.     Technical  evaluations  of 
testing  laboratories  are  made  to  determine  if  tliey  have  the 
capacity  to  carry  out  properly  the  tests  specified  in  the 
product  performance  standards.     ijontechnical  evaluations  which 
are  also  appropriate  and  have  been  mentioned  earlier  are 
managerial  independence  from  manufacturers  and  vendors,  and 
recoras  keeping.     /another  nontechnical  criterion  which  may  be 
relevant  concerns  equal  employment  opportunity    (LLC)    at  the 
testing  laboratories.     Tlie  relevance  of  this  criterion  is  a 
policy  question  best  answered  by  i:iLLCJ. 

The  checking  of  laboratories  to  determine  whether  the  pre- 
stated  criteria  are  satisfied  can  be  considered  to  be  a  three- 
step  process--examination  and  evaluation  by  LLSL  followed,  if 
merited,   by  ImILLCJ  accreditation.     Laboratory  e:caminations 
provide  the  information  upon  which  laboratory  evaluations  are 
made,  v;h.ich  in  turn  serve  as  the  basis  for  LLSL '  s 
reccmraendation  to  LILLCJ  for  laboratory  accreditation.  On-site 
examinations  at  the  laboratories  are  expensive  in  term.s  of 
required  travel  and  mcanpower .     To  the  extent  practical, 
examinations  will  involve  the  laboratories  in  question  sending 
informLation  to  LLSL  for  evaluation.     Lcwever ,   a  few  on-site 


exaiainations  vvill  have  Lo  be  made,   some  on  an  unannounced 
basis,   to  check  the  information  furnished  by  the  laboratories. 


Laboratories  will  be  evaluated  and  accredited  on  a 
standard-by-standard  basis.     Isolation  of  the  specific  causes 
of  problems  being  experienced  by  laboratories  failing  to  meet 
certain  performance  criteria  will  be  outside  the  scope  of  this 
program  unless  a  problem  is  widespread  among  testing 
laboratories  involved  with  the  Compliance  Testing  System,  or 
the  basis  for  the  problem  can  be  established  with  very  minor 
effort. 

Compliance  Information  System 

A  "Compliance  Information  System"  is  needed  as  part  of  the 
Compliance  Testing  System  for  the  purpose  of  eliciting 
information  concerning  the  performance  of  devices  listed  on  the 
l.ILLCJ  qualified  products  lists.     Reports  will  be  tabulated 
from  laboratories  performing  qualification  and/or  acceptance 
testing.     Cooperation  in  supplying  this  information  will  be  a 
condition  for  being  an  accredited  laboratory.     In  addition, 
field  performance  data  will  be  solicited  from  law  enforcement 
agencies  using  the  equipment  covered  by  KILECJ  standards. 

The  Comipliance  Information  System  v/ill  help  to  assure  that 
(a)   equipment  listed  on  a  IjILLCJ  qualified  products  list 
continues  to  perform  in  accordance  with  the  requirem.ents  of  the 
IJILECJ  standard,    (b)   experience  in  field  use  does  not  indicate 
an  excessive  breakdown  rate  or  the  existence  of  m.aintenance 
problems,    (c)   any  need  for  revision  of  the  standard  is 
recognized,  and   (d)   problems  in  the  performance  of  any  of  the 
testing  laboratories  are  recognized.     V7hen  tabulated 
information  indicates  the  need,  a  special  investigation  will  be 
made.     If,   for  example,  it  is  determined  that  a  manufacturer 
has  a  quality  control  problem,  continued  listing  of  his  product 
on  the  QPL  will  be  reviewed.     If  an  excessive  field  breakdown 
or  maintenance  problem  is  found,  either  the  product  will  be 
removed  from  the  QPL  or  the  applicable  standard  will  be 
revised,  as  appropriate.     If  a  problem  in  testing  laboratory 
performance  is  found,   the  continued  accreditation  of  that 
laboratory  will  be  reviewed. 

Ihe  National  Highway  Traffic  Safety  Administration   (NIITSA) , 
Department  of  Transportation,  announced  in  the  Ilovember  5, 
1973,   issue  of  the  Federal  Register,  page  30159,  the 
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establishment  of  a  program  having  a  Standards  Compliance 
Information  System  and  many  other  features  similar  to  those 
recommended  for  the  NILECJ  Compliance  Testing  System.  The 
NHTSA  Compliance  Testing  System  was  planned  with  NBS/LESL 
consultation;  the  breath  alcohol  testing  equipment  performance 
standards  for  which  it  was  established  are  being  developed  for 
NHTSA  by  LESL.     It  should  be  noted  that  only  equipment  on  the 
UHTSA  qualified  products  list  may  be  purchased  with  NHTSA 
funds  —  a  policy  which  NILECJ  may  also  wish  to  adopt. 

Role  and  Qualifications  of  NBS 

The  large  amount  of  both  product  qualification  and 
acceptance  testing  necessitates  the  involvement  of  testing 
laboratories  other  than  NBS.     NBS/LESL 's  role  is  in  the 
development  of  the  standards,  it^  the  arbitration — if  needed — of 
any  disputes  which  may  arise  concerning  compliance  test 
results,  and  in  the  management  of  the  compliance  testing 
system,  but  not  in  compliance  testing  per  se.     All  of  these 
activities  will  of  course  be  under  the  direction  of  NILECJ. 

Each  QPL  will  be  a  NILECJ  document.     Its  content  will  be 
based  largely  on  the  information  obtained  from  the  Compliance 
Testing  System.     Likewise,  the  official  list  of  accredited 
laboratories  will  be  a  NILECJ  document,  although  the 
accreditations  will  be  based  on  evaluations  performed  by 
NBS/LESL. 

NBS  has  had  and  continues  to  have  considerable  involvement 
with  laboratory  evaluations.     Current  NBS  projects  involving 
laboratory  evaluation  relate  to  the  testing  of  cement  and 
concrete,  paper  and  paper  boxes,  rubber,  color  and  appearance, 
mobile  homes,  mass  standards,  state  metrology  laboratories, 
clinical  chemistry  and  toxicology.     A  special  Laboratory 
Performance  Technology  Section  has  been  established  at  NBS. 
NBS  management,  together  with  other  officials  in  the  U.S. 
Department  of  Commerce,  are  planning  with  industry  and 
professional  associations  the  establishment  of  a  national 
system  for  laboratory  evaluation;  several  public  hearings  have 
been  held  on  the  subject  at  NBS. 


Although  i>iijS  has  almost  no  regulatory  authority,   it  has 
established  a  reputation  of  being  a  fair  and  careful  arbitrator 
iii  disputes  involving  measurement.     Members  of  the  UBS  staff 
liave  been  among  the  vanguard  in  the  development  of  improved 
techniques  for  laboratory  evaluation.     An  Association  of 
Official  Analytical  Chemists  publication,  "Statistical 
Techniques  for  Collaborative  Tests,"  v/as  written  by  an  L'BS 
statistician,  Dr.  Vv.  J.   Youden.     Chapter  three  of  UBS  Special 
Publication  300  titleu  " Interlaboratory  Tests"  is  a  collection 
of  papers  on  the  subject  by  UBS  authors.     The  editor  of  the 
"ASTM  bulletin"   included  the  following  quotation  in  his 
introductory  remarks  to  a  paper  about  interlaboratory  testing 
by  two  liBS  scientists:     "This  paper  gives  a  very  complete 
treatment  of  the  problem  which  almost  every  ASTM  committee  is 
constantly  trying  to  solve  .    .    .    (it  is)   a  more  comprehensive 
approach  to  the  problem  of  designing  and  interpreting 
interlaboratory  studies  than  has  appeared  in  the  literature  up 
to  now."      (ASTM  Bulletin,  July  1959,  page  53). 

During  the  development  of  performance  standards,  LESL  must 
do  a  certain  amount  of  product  testing  to  determine  current 
performance  levels.     This  work  is  rarely  sufficient  to  negate 
the  need  for  qualification  tests  after  promulgation  of  the 
standard,  but  pertinent  test  results  obtained  during  the 
development  of  the  standard  should  certainly  be  utilized  by  the 
Compliance  Testing  System..     Conversely,   the  compliance  test 
results  vvill  serve  as  early  indicators  that  particular  IJILECJ 
standards  should  be  considered  for  revision.     Thus,  close 
liaison  between  the  standards  developm.ent  program  and  the 
Compliance  Testing  Program  will  be  essential.     This  two-way 
flow  of  information  will  be  enhancea  by  both  programs  being 
within  L£SL. 

Who  Pays? 

Uho  should  pay  the  costs  of  a  national  compliance  testing 
system  for  law  enforcement  equipment?     Let  us  consider 
separately  each  of  the  component  parts  of  a  complete, 
coiuprehensive  system. 

Laboratory  Lvaluation  and  Accreditation 

In  and  of  itself,  a  laboratory  evaluation  and  accreditation 
program  is  of  no  value.  Such  a  program  is,  however,  central  to 
any  coiupliaiice  testing  systen,  no  matter  hov;  sim.ple  or  complex, 
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large  or  snail.     I.one  of  the  otlicr  possible  components  of  gucI^ 
a  systen-,  can  function  without  the  availability  of  testinq 
laboratories  v;hose  test  results  are  accepted  by  both  the  buyer 
and  seller.     These  considerations  under ly  the  question  of  v.-ho 
should  pay  for  laboratory  evaluation  and  accreditation. 

In  a  mature,   functionincj  corr.pliance  testincj  system,  we 
recouuuend  that  most,   if  not  all,  of  the  full  pro-rata  cost  of 
evaluating  and  accrediting  a  testing  laboratory  be  borne  by 
tiiat  laboratory  itself.     In  a  competitive  environment,  this 
would  assure  that  accreditation  would  be  sought  by  only  those 
testing  laboratories  to  wliom  it  was  financially  attractive. 
Clearly,   this  would  be  the  most  cost-effective  and  equitable 
arrangement . 

In  starting  up  such  a  program,   however,  when  it  is  not 
clear  how  much  testing  business  will  be  generated,   and  who  the 
customers  will  be,   it  would  be  unreasonable  and  unrealistic  to 
expect  the  testing  laboratories  to  bear  the  costs  of  their 
evaluation  and  accreditation,  plus  the  program  start-up  costs. 
It  is  recomraenaed  tliat  the  sponsor  of  the  program,   iJILLCJ,  pay 
the  major  portion  or  all  of  these  costs,   in  order  to  get  the 
prograr.i  under  way  and  to  attract  as  many  testing  laboratories 
as  possible  into  the  program.     It  is  further  recommended  that 
the  proportion  of  the  costs  borne  by  the  testing  laboratories 
themselves  be  increased  gradually,  but  as  rapidly  as  the 
circumstances  warrant,   until  the  program  is  fully  self- 
supporting,   or  nearly  so.     It  must  be  realized  that,   for  each 
equipment  item  in  the  program,  at  least  one  testing  laboratory 
must  obtairi  enough  business  to  justify  its  cost  of  obtaining 
and  maintaining  accreditation.     'io  assure  this,   some  continuing 
HILLCJ  subsidy  may  be  required,   in  particular  cases. 

Manufacturer  Certification  Based  on  Accredited 
Laboratory  Testing 

Under  this  system  option,  once  one  or  more  testincj 
laboratories  are  accredited  to  test  an  equipment  item  in 
accordance  with  the  appropriate  rJILL;CJ  standard,  any  and  all 
manufacturers  of  that  item  would  be  free  to  contract 
individually  with  the  accredited  testing  laboratory  of  their 
choice,  at  a  fee  agreed  to  by  them,   to  have  their  products 
tested  for  compliance.     The  manufacturers  could  use  these  test 
results  in  their  advertising  and/or  marketing,  could  use  them 
as  a  basis  for  certifying  Uiat  their  products  complied  with  the 
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appropriate  standard,  anu  could  therefore  bid  on  procurement 
contracts  that  required  compliance  with  the  IIILLCJ  standard. 

Vvhile  sorae  rationale  exists  for  LILECJ  financial  support  of 
this  program  option,  no  simple  means  is  envisioned  for  doing 
so.     It  is  therefore  recommended  that  payment  for  services 
under  this  program  be  fully  borne  by  the  individual 
manufacturer  or  distributor.     Ihe  operation  of  this  systen 
would  involve  little  or  no  Government  participation;   it  would 
probably  be  little  used  if  Qualified  Products  Lists  were 
establishea    (see  below) . 

Routine  Acceptance  Testing 

One  of  the  options  available  to  a  purchaser,  after  he  has 
received  a  shipment  of  material  claimed  to  comply  v/ith  the 
requirements  of  a  i.ILLCJ  standard,   is  to  verify  its  compliance 
by  having  tiie  shipment  tested  by  a  qualified  testing 
laboratory.     uhere  such  acceptance  testing  is  desired,  it 
should  be  performed  on  a  f ee-f or-service  basis,  entirely  paid 
for  by  the  purchaser  and/or  by  the  supplier,   in  accordance  v;ith 
the  requiremerit  of  their  purchase  agreement. 

It  is  recoruucrided  that  the  States  and  local  jurisdictions 
oe  authorized  and  encouraged,  but  not  required,   to  [jay  for  such 
acceptance  tests  v/ith  LL7iA  block-grant  or  other  LFAA  funds 
allocated  to  then.     This  authorisation  should  govern  whether  or 
not  the  equipment  itself  is  paia  for  with  Lr:AA  funds. 

,  Qualified  Products  lists 

The  existence  of  an  official  MLI'CJ  Qualified  Products  List 
for  a  particular  equipm.ent  itoni  would  piakc  it  very  easy  for  a 
purchaser  to  acquire  I.iILi;CJ-Gtai!card  equipment.     In  sone  cases, 
such  as  v;hen  the  cost  of  testing  was  large  compared  to  the  cost 
of  the  shipment,   reference  to  a  QPL  v/ould  be  perhaps  the  only 
way  in  which  an  objective,   cost-effective  purchase  could  be 
made . 

The  establishment  and  maintenance  of  a  QPL  for  each 
appropriate  equipment  item  is  thus  a  major  tool  whereby  I.ITiECJ 
could  assure  the  quality  of  equipment  procured  by  State  and 
local  agencies.     Similarly,   the  listing  of  its  product  on  a 
iJILLCJ  QPL  could  be  of  substantial  economic  benefit  to  an 
equipment  manufacturer.     Joint  funding  of  a  QPL  by  L'lLLCJ  and 
the  interested  manuf actuorer s  is  therefore  warranted,   and  is 
recomraended . 

A  OPL  program  sJiould  be  run  Ijv  LEAA/!JILECJ  itself,    or  on 
its  I)ehalf  by  another  public  agency. 
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GLOSSARY 

This  Glossary  is  intended  to  assist  the  reader  understand 
the  terminology  by  comparing  and  contrasting  related  terms. 
The  entries  are  grouped  into  three  sections  relating  to  the 
subject  areas  of  product  standards,  product  evaluation  and 
laboratory  accreditation.     Related  entries  within  each  section 
are  also  grouped  rather  than  listed  alphabetically.  However, 
an  alphabetical  listing  of  entries  is  more  convenient  for 
locating  the  definition  of  a  specific  term.     Hence,  a  separate 
Index  of  Terms  is  included  at  the  back  of  the  Glossary. 

Many  of  the  definitions  in  this  Glossary  are  based  on 
definitions  in  an  unpublished  document  drafted  earlier  at  NBS. 
Some  terms  in  that  document  are  included  in  this  glossary  even 
though  they  are  not  used  in  this  report  concerning  compliance 
testing.     Terms  such  as,   "mandatory  standard",  "quasi-mandatory 
standard",  and  "consensus  standard"  are  of  general  interest  to 
people  working  with  standards  and  thus  were  included  in  this 
Glossary. 
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1,      PRODUCT  STANDARDS 


1.1  STANDARD 

A  prescribed  set  of  conditions  or  requirements — the 
physical,   functional,  performance,  or  conformance 
characteristic  thereof — to  be  satisfied  by  a  product. 
Standards  included  under  this  definition  are  called  product 
standards  or  engineering  standards. 

1.1.1  VOLUNTARY  STANDARD 

A  standard  with  which  there  is  no  obligation  to  comply.  A 
voluntary  standard  may  become  a  quasi-mandatory  standard, 
mandatory  standard,  code,  regulation,  or  rule  as  a  result  of 
utilization  or  adoption  by  a  regulatory  authority. 

1.1.2  QUASI -MANDATORY  STANDARD 

A  Standard  with  which  there  is  no  legal  obligation  to 
comply,  but  which  is  required  in  practice  or  under  certain 
conditions,   such  as  a  requirement  of  a  marketplace  or 
compatibility  with  other  products. 

1.1.3  MANDATORY  STANDARD 

A  Standard  with  which  there  is  an  obligation  to  comply  by 
virtue  of  an  action  by  government  or  by  an  authority  endowed 
with  the  necessary  legal  power;  called  a  code,  regulation,  or 
rule . 

1.1.4  CONSENSUS  STANDARD 

A  standard  for  which  there  is  general  agreement  among  those 
affected  by  the  standard  that  the  prescribed  set  of  conditions 
and  requirements  are  technically  sound  and  meet  the  needs 
prevailing  at  that  time. 

1.1.5  PERFORMANCE  STANDARD 

A  standard  which  prescribes  the  acceptable  functional  or 
operational  characteristics  of  a  material,  product  or  system  in 
accordance  with  the  use  to  which  the  performance  would  apply; 
includes  or  references  the  test  methods  by  which  these 
characteristics  are  measured. 
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1.1.6     DESIGN  STANDARD 


A  Standard  which  describes  the  required  physical  or 
dimensional  characteristics  of  a  product  or  system  and,   in  some 
cases,   its  manufacture,  construction  or  fabrication. 

1.1.7     NILECJ  STANDARD 

A  voluntary  standard  developed  by  the  Law  Enforcement 
Standards  Laboratory  at  the  National  Bureau  of  Standards  and 
promulgated  by  the  National  Institute  of  Law  Enforcement  and 
Criminal  Justice   (NILECJ),  U.S.  Department  of  Justice,  for 
equipment  used  in  lav/  enforcement,  crime  deterrence,  or 
criminal  justice  activities.     NILECJ  Standards  are  technical 
documents  consisting  of  performance  and  other  requirements 
together  with  a  description  of  test  methods. 

1.2  TEST  METHOD 

A  description  of  the  test  procedures,  equipment  and 
methodology  for  testing  a  material  or  product  in  determining 
its  conformance  to  a  standard  or  other  set  of  conditions  and 
requirements . 

1.3  REFERENCE  I^TERIAL 

A  material,  substance,  or  device  whose  intrinsic  properties 
are  used  for  physical  comparison   (e.g.,  an  NBS  Standard 
Reference  Material   (SRM)  which  is  a  well  characterized  and 
certified  material  produced  in  quantity  and  used  to  develop 
reference  methods  of  analysis  and  tests  and  to  calibrate 
measurement  systems),  sometimes  referred  to  as  a  standard. 


2.      PRODUCT  EVALUATION 

2.1     PRODUCT  COMPLIANCE  TEST 

A  test  to  determinate  the  design  or  performance 
characteristics  of  products  for  the  purpose  of  establishing 
their  conformance  with  the  requirements  of  applicable 
standards,  codes  or  other  requirements. 


2.1.1     PRODUCT  ACCEPTANCE  TEST 


A  compliance  test  to  determine  the  acceptability  of 
delivered  items  which  have  been  purchased  under  a  contract 
requiring  compliance  with  the  appropriate  standard,  code,  or 
other  requirement. 

2.1.2     PRODUCT  QUALIFICATION  TEST 

A  compliance  test  performed  in  advance  of  and  independent 
of  any  specific  procurement  action,  for  the  purpose  of 
establishing  a  qualified  products  list  or  authorizing  the  use 
of  a  logo. 

2.2  QUALIFIED  PRODUCTS  LIST  (QPL) 

A  list  of  products,  and  their  manufacturer,  which  have  been 
tested  and  found  to  comply  with  the  requirements  of  applicable 
standards,  codes  or  other  requirements. 

2 . 3  LOGO 

A  symbol,  label,  hallmark  or  statement  authorized  by  a 
certification  agency  for  affixing  to  a  product  indicating  that 
the  product  is  in  conformance  v/ith  specified  provisions  of  the 
applicable  standard,  code  or  other  requirement. 


,   ;  3.      LABORATORY  ACCREDITATION 

3 . 1  ACCREDITATION 

The  act  of  giving  official  recognition  of  specific 
qualifications.     Accreditation  of  a  laboratory  indicates  that 
there  is  no  apparent  reason  why  the  laboratory  cannot  perform  a 
function  adequately.     It  does  not  guarantee  the  quality  of 
future  performance. 

3.2  CERTIFICATION 

The  act  of  giving  an  official  warranty  or  guarantee  of 
performance.     There  is  no  valid  approach  to  examining  a 
laboratory  for  laboratory  certification  because  guarantees  of 
future  performance  cannot  be  made. 

3.3  TESTING  LABORATORY 

A  place  equipped  and  staffed  to  conduct  product  tests 
including  product  compliance  tests. 
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3.3.1  INDEPENDENT  TESTING  LABORATORY 

A  testing  laboratory  which  has  no  organizational  tie  or 
financial  interest  in  a  manufacturer,  vendor,  or  the  promotion 
of  a  specific  product  on  which  tests  are  performed.     It  has 
sufficient  breath  of  activity  so  that  the  loss  or  award  of  a 
specific  contract  for  test  services  would  not  be  a  substantive 
factor  in  the  financial  well-being  of  the  laboratory.     It  may 
offer  test  services  under  contract  or  on  a  fee  basis,  and  may 
be  a  profit  or  non-profit  organization. 

3.3.2  CORPORATE  OR  CAPTIVE  TESTING  LABORATORY 

A  testing  laboratory  organizationally  affiliated  with  a 
product  manufacturer  or  vendor. 

3.4  LABORATORY  EXAMINATION 

The  process  of  obtaining  information  in  order  to  judge  the 
laboratory's  capability  for  performing  specified  product  tests, 
e.g.,  product  compliance  tests.     Laboratory  examinations 
provide  information  upon  which  laboratory  evaluations  are  made. 
Laboratory  examinations  can  involve  on-site  and  remote 
activities;   the  remote  activities  involve  the  laboratories 
sending  information  to  the  evaluation  agency. 

3.5  LABORATORY  EVALUATION 

The  decision  making  process,  using  information  from 
laboratory  examinations,  upon  which  laboratory  accreditation — 
or  a  recommendation  for  laboratory  accreditation — is  based. 

3.5.1     LABORATORY  EVALUATION  CRITERIA 

Statements  prescribing  the  organizational  and  technical 
resources,  the  equipment  and  facilities,   the  operational 
procedures,  and  the  minimum  technical  performance  levels 
required  of  a  testing  laboratory  for  accreditation. 

3.6  COLLABORATIVE  REFERENCE  TESTING 

A  program  or  system  in  which  a  uniform  material  or  product, 
properly  randomized  or  of  controlled  characteristics,  is 
distributed  to  participating  laboratories.     The  laboratories 
test  the  material  or  product  and  submit  the  test  results  to  the 
agency  administering  the  collaborative  test  for  evaluation. 
Collaborative  reference  testing  may  be  used  voluntarily  by 
testing  laboratories  for  self -evaluating  or  improving  test 
criteria  and  methodology.   In  conjunction  with  other  procedures, 
collaborative  reference  testing  may  be  used  in  laboratory 
examination  for  accreditation  or  continued  accreditation. 
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